Accelerated Extra-Gradient Descent: A Novel Accelerated First-Order Method

We provide a novel accelerated first-order method that achieves the asymptotically optimal con-vergence rate for smooth functions in the first-order oracle model. To this day, Nesterov’s AcceleratedGradient Descent (agd) and variations thereof were the only methods achieving acceleration in thisstandard blackbox model. In contrast, our algorithm is significantly different from agd, as it relies ona predictor-corrector approach similar to that used by Mirror-Prox [11] and Extra-Gradient Descent [7]in the solution of convex-concave saddle point problems. For this reason, we dub our algorithmAccelerated Extra-Gradient Descent (axgd).Its construction is motivated by the discretization of an accelerated continuous-time dynamics [8]using the classical method of implicit Euler discretization. Our analysis explicitly shows the effects ofdiscretization through a conceptually novel primal-dual viewpoint. Finally, we present experimentsshowing that our algorithm matches the performance of Nesterov’s method, while appearing morerobust to noise in some cases.